12. A/B Testing

A/B Testing Heading

A/B testing

ND036 C4 L04 A08 A-B Testing

A/B Testing Graphic

A/B Testing Recap

New Terms

A/B testing: multi-variant testing where we compare one version of a product with another version of the same product. For example, we can A/B test text on a button on our page and see which text gets more users to click on the button.

Things to Remember About A/B testing

  • Why is A/B testing so important? We don’t always want to risk and change our product without knowing if the change is really a good one. It is especially important when we are testing a major change.

  • Ask for help from your Data Analysts when possible. They can confirm the validity of the test.

A/B Testing Quiz

Our competitor changed all their buttons from green to blue, your VP wants you to change our buttons as well. What should you do?
SOLUTION: You suggest A/B testing, where we change the button color for 5% of our users and see how that performs before we make a decision.

A/B Testing Goal Quiz

What is the goal of running an A/B test?
SOLUTION: To compare one version of a flow in your product to another version and see which performs better

A/B Testing Problem Set

In order to create a good A/B test you will need to set
SOLUTION:
  • Well-defined options for each arm
  • A Hypothesis about the results
  • The minimum number of responses required for validity

A/B Testing Further Research

Additional Reading

Learn more about A/B Testing here:

Next Concept